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Abstract 

Gossip algorithms are attractive for in-network processing in sensor networks because they do not require any 
specialized routing, there is no bottleneck or single point of failure, and they are robust to unreliable wireless 
network conditions. Recently, there has been a surge of activity in the computer science, control, signal processing, 
and information theory communities, developing faster and more robust gossip algorithms and deriving theoretical 
performance guarantees. This article presents an overview of recent work in the area. We describe convergence rate 
results, which are related to the number of transmitted messages and thus the amount of energy consumed in the 
network for gossiping. We discuss issues related to gossiping over wireless links, including the effects of quantization 
and noise, and we illustrate the use of gossip algorithms for canonical signal processing tasks including distributed 
estimation, source localization, and compression. 

I. Introduction 

Collaborative in-network processing is a major tenet of wireless sensor networking, and has received much 
attention from the signal processing, control, and information theory communities during the past decade |[l|. Early 
research in this area considered applications such as detection, classification, tracking, and pursuit ||2)-||5). By 
exploiting local computation resources at each node, it is possible to reduce the amount of data that needs to be 
transmitted out of the network, thereby saving bandwidth and energy, extending the network lifetime, and reducing 
latency. 
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In addition to having on-board sensing and processing capabilities, the archetypal sensor network node is battery- 
powered and uses a wireless radio to communicate with the rest of the network. Since each wireless transmission 
consumes bandwidth and, on common platforms, also consumes considerably more energy than processing data 
locally ||6), ||7|, reducing the amount of data transmitted can significantly prolong battery life. In applications where 
the phenomenon being sensed varies slowly in space, the measurements at nearby sensors wUl be highly correlated. 
In-network processing can compress the data to avoid wasting transmissions on redundant information. In other 
applications, rather than collecting data from each node, the goal of the system may be to compute a function of 
the data such as estimating parameters, fitting a model, or detecting an event. In-network processing can be used 
to carry out the computation within the network so that, instead of transmitting raw data to a fusion center, only 
the results of the computation are transmitted to the end-user. In many situations, in-network computation leads to 
considerable energy savings over the centralized approach (8), |[9). 

Many previous approaches to in-network processing assume that the network can provide specialized routing 
services. For example, some schemes require the existence of a cyclic route through the network that passes 
through every node precisely one tim^*] ||9)-|[TT). Others are based on forming a spanning tree rooted at the fusion 



center or information sink, and then aggregating data up the tree |]8], p2) , |13|. Although using a fixed routing 
scheme is intuitive, there are many drawbacks to this approach in wireless networking scenarios. Aggregating data 
towards a fusion center at the root of a tree can cause a bottleneck in communications near the root and creates a 
single point of failure. Moreover, wireless links are unreliable, and in dynamic environments, a significant amount 
of undesirable overhead traffic may be generated just to establish and maintain routes. 



A. Gossip Algorithms for In-Network Processing 

This article presents an overview of gossip algorithms and issues related to their use for in-network processing 
in wireless sensor networks. Gossip algorithms have been widely studied in the computer science community for 
information dissemination and search |fT4|-p6|. More recently, they have been developed and studied for information 
processing in sensor networks. They have the attractive property that no specialized routing is required. Each node 
begins with a subset of the data in the network. At each iteration, information is exchanged between a subset of 
nodes, and then this information is processed by the receiving nodes to compute a local update. 

Gossip algorithms for in-network processing have primarily been studied as solutions to consensus problems, 
which capture the situation where a network of agents must achieve a consistent opinion through local information 
exchanges with their neighbors. Early work includes that of Tsitsiklis et al. p7) , | [T8| . Consensus problems have 
arisen in numerous applications including: load balancing p9| ; alignment, flocking, and multi-agent collabora- 
tion pO) , pTj ; vehicle formation |22|, tracking and data fusion p3) , and distributed inference p4) . 

The canonical example of a gossip algorithm for information aggregation is a randomized protocol for distributed 
averaging. The problem setup is such that each node in a rt-node network initially has a scalar measurement value, 

'This is a Hamiltonian cycle, in graph-theoretic terms. 
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and the goal is to have every node compute the average of all n initial values - often referred to as the average 
consensus. In pairwise randomized gossiping p5) , each node maintains an estimate of the network average, which 
it initializes with its own measurement value. Let x{t) denote the vector of estimates of the global averages after 
the i*'* gossip round, where a;(0) is the vector of initial measurements; that is, Xi{t) is the estimat^at node i after 
t iterations. In one iteration, a randomly selected pair of neighboring nodes in the network exchange their current 
estimates, and then update their estimates by setting Xi{t + 1) = Xj{t + 1) = {xi{t) + Xj{t)) /2. A straightforward 
analysis of such an algorithm shows that the estimate at each node are guaranteed to converge to the average, 
Xave = ^X]r=i^«(0)' ^'^^E the network is connected (information can flow between all pairs of nodes), 
and as long as each pair of neighboring nodes gossips frequently enough; this is made more precise in Section |ll] 
below. Note that the primitive described above can be used to compute any function of the form X]i"=i fi{^i{^)) 
by properly setting the initial value at each node, and while this is not the most general type of query, many useful 
computations can be reduced in this form as will further be highlighted in Sections IV and |V] 



Gossip algorithms can be classified as being randomized or deterministic. The scheme described above is 
randomized and asynchronous, since at each iteration a random pair of nodes is active. In deterministic, synchronous 
gossip algorithms, at each iteration node i updates Xi{t + 1) with a convex combination of its own values and the 



values received from all of its neighbors, e.g., as discussed in |26|. Asynchronous gossip is much better suited 
to wireless sensor network applications, where synchronization itself is a challenging task. Asynchronous gossip 
can be implemented using the framework described in p8) , | |27| . Each node runs an independent Poisson clock, 
and when node i's clock "ticks", it randomly selects and gossips with one neighbor. In this formulation, denoting 
the probability that node i chooses a neighbor j by Pij, conditions for convergence can be expressed directly 
as properties of these probabilities. Gossip and consensus algorithms have also been the subject of study within 
the systems and control community, with a focus on characterizing conditions for convergence and stability of 
synchronous gossiping, as well as optimization of the algorithm parameters Pij; see the excellent surveys by 
Olfati-Saber and Murray |281, and Ren et al. p9| , and references therein. 

B. Paper Outline 

Our overview of gossip algorithms begins on the theoretical side and progresses towards sensor network appli- 
cations. Each gossip iteration requires wireless transmission and thus consumes valuable bandwidth and energy 
resources. Section |ll] discusses techniques for bounding rates of convergence for gossip, and thus the number of 
transmissions required. Because standard pairwise gossip converges slowly on wireless network topologies, a large 
body of work has focused on developing faster gossip algorithms for wireless networks, and this work is also 
described. When transmitting over a wireless channel, one must also consider issues such as noise and coding. 



Section III discusses the effects of finite transmission rates and quantization on convergence of gossip algorithms. 



Finally, Section IV illustrates how gossip algorithms can be applied to accomplish distributed signal processing 



^Throughout, we will sometimes alternatively refer to the estimates Xi (t) as states, and to nodes as agents. 
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tasks such as distributed estimation and compression. 

II. Rates of Convergence and Faster Gossip 

Gossip algorithms are iterative, and the number of wireless messages transmitted is proportional to the number 
of iterations executed. Thus, it is important to characterize rate of convergence of gossip and to understand what 
factors influence these rates. This section surveys convergence results, describing the connection between the rate 
of convergence and the underlying network topology, and then describes developments that have been made in the 
area of fast gossip algorithms for wireless sensor networks. 



A. Analysis of Gossip Algorithms 

In pairwise gossip, only two nodes exchange information at each iteration. More generally, a subset of nodes may 
average their information. All the gossip algorithms that we will be interested in can be described by an equation 
of the form 

x{t + l) = W{t)x{t), (1) 

where W{t) are randomly selected averaging matrices, selected independently across time, and x{t) E M" is the 
vector of gossip states after t iterations. When restricted to pairwise averaging algorithms, in each gossip round 



only the values of two nodes i,j are averaged (as in |25|) and the corresponding W{t) matrices have 1/2 in the 
coordinates {i, i), {j, i), and a diagonal identity for every other node. When pairwise gossip is performed 
on a graph G = {V,E), only the matrices that average nodes that are neighbors on G (i.e., i.j £ E) are selected 
with non-zero probability. More generally, we will be interested in matrices that average sets of node values and 
leave the remaining nodes unchanged. A matrix W{t) acting on a vector x{t) is set averaging matrix for a set S 
of nodes, if 

and Xi{t + 1) = Xi{t), i ^ S. Such matrices therefore have entry 1/\S\ at the coordinates corresponding to the set 
S and a diagonal identity for all other entries. 

It is therefore easy to see that all such matrices will have the following properties: 

(3) 

W{t)l^l, 

which respectively ensure that the average is preserved at every iteration, and that 1, the vector of ones, is a fixed 
point. Further, any set averaging matrix W is symmetric and doubly stochastic. A matrix is doubly stochastic if its 
rows sum to unity, and its columns also sum to unity, as implied in (|3]l. The well-known Birkhoff-von Neumann 
Theorem states that a matrix is doubly stochastic if and only if it is a convex combination of permutation matrices. 
In the context of gossip, the only permutation matrices which contribute to the convex combination are those which 
permute nodes in S to other nodes in S, and keep all other nodes not in S fixed. The matrix W must also be a 
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projection matrix; i.e., = W since averaging the same set twice no longer changes the vector x{t). It then 
follows that W must also be positive-semidefinite. 

We are now ready to understand the evolution of the estimate vector x{t) through the product of these randomly 
selected set averaging matrices: 

t 

x{t + l) ^W{t)x{t) ^W^W(k) x{0). (4) 

fc=0 

Since W{t) are selected independently across time, EVK(t) = EVF(O) and we can drop the time index and 
simply refer to the expected averaging matrix EW^, which is the average of symmetric, doubly stochastic, positive 
semidefinite matrices and therefore also has these properties. The desired behavior is that x{t + 1) — > Xave^ that is 
equivalent to asking that 

T\w{k)^-n^. (5) 

k=0 

B. Expected behavior 

We start by looking at the expected evolution of the random vector x{t) by taking expectations on both sides of 

Ea;(t+1) =E j J]^ VK(fc) j x{Q) ^ {EWy+^x{0), (6) 
\fc=o / 

where the second equality is true because the matrices are selected independently. Since ¥W is a convex combination 
of the matrices W{t) which all satisfy the conditions ([3]), it is clear that WW is also a doubly stochastic matrix. We 
can see that the expected evolution of the estimation vector follows a Markov chain that has the Xavcl vector as its 
stationary distribution. In other words, 1 is an eigenvector of EVK with eigenvalue 1. Therefore if the Markov chain 
corresponding to WW is irreducible and aperiodic, our estimate vector will converge in expectation to the desired 
average. Let A2(E[VK]) be the second largest eigenvalue of WW. If condition ([3| holds and if A2(E[M^]) < 1, 
then x{t) converges to Xave^ in expectation and in mean square. Further precise conditions for convergence in 
expectation and in mean square can be found in pO). 



C. Convergence rate 

The problem with the expectation analysis is that it gives no estimate on the rate of convergence, a key parameter 
for applications. Since the algorithms are randomized, we need to specify what we mean by convergence. One notion 
that yields clean theoretical results involves defining convergence as the first time where the normalized error is 
small with high probability, and controlling both error and probability with one parameter, e. 

Definition 1: e-averaging time Tayei^)- Given e > 0, the e-averaging time is the earliest gossip round in which 
the vector x{k) is e close to the normalized true average with probability greater than 1 — e: 

r_(6)=sup inf |p f ^^^^-^^ > 6^ < e| . (7) 
.(o)*=0'i.2... I \^ ||x(0)|| J J 

Observe that the convergence time is defined for the worst case over the initial vector of measurements a::(0). This 

definition was first used in p5[ (see also fTl\ for a related analysis). 
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The key technical theorem used in the analysis of gossip algorithms is the following connection between the 
averaging time and the second largest eigenvalue of EVF: 

Theorem 1: For any gossip algorithm that uses set-averaging matrices and converges in expectation, the averaging 
time is bounded by 



This theorem is a slight generalization of Theorem 3 from p5) for non-pairwise averaging gossip algorithms. There 
is also a lower bound of the same order, which implies tha|^Ta„e(e, EW') = e(loge-V(l - X^iVW))). 

The topology of the network influences the convergence time of the gossip algorithm, and using this theorem 
this is precisely quantified; the matrix E[VF] is completely specified by the network topology and the selection 
probabilities of which nodes gossip. The rate at which the spectral gap 1 — A2(]E[iy]) approaches zero, as n 
increases, controls the e-averaging time Tave- The spectral gap is related to the mixing time (see, e.g., |32|) of a 
random walk on the network topology. Roughly, the gossip averaging time is the mixing time of the simple random 
walk on the graph times a factor of n. One therefore would like to understand how the spectral gap scales for 
different models of networks and gossip algorithms. 



This was first analyzed for the complete graph and uniform pairwise gossiping p3) , p5) , 1 30 1. For this case it was 
shown that A2(IE[W^]) = 1 — 1/n and therefore, Tave = 0(nloge~^). Since only nearest neighbors interact, each 
gossip round costs two transmitted messages, and therefore, 0(nloge^^) gossip messages need to be exchanged to 
converge to the global average within e accuracy. This yields 0(nlogn) messages to have a vanishing error with 
probability 1/n, an excellent performance for a randomized algorithm with no coordination that averages n nodes 
on the complete graph. For other well connected graphs (including expanders and small world graphs), uniform 
pairwise gossip converges very quickly, asymptotically requiring the same number of messages (0(nloge~^)) as 
the complete graph. Note that any algorithm that averages n numbers with a constant error and constant probability 
of success should require messages. 

If the network topology is fixed, one can ask what is the selection of pairwise gossiping probabilities that 
maximizes the convergence rate (i.e. maximizes the spectral gap). This problem is equivalent to designing a Markov 
chain which approaches stationarity optimally fast and, interestingly, it can be formulated as a semidefinite program 

'Because our primary interest is in understanding scaling laws — how many messages are needed as the network size grows — our discussion 
centers on the order-wise behavior of gossip algorithms. Recall the Landau or "big O" notation: a function / is asymptotically bounded above 
by g, written f(n) = 0(g{n)), if there exist constants A'^ > and ci > such that /(n) < ci g(n) for all n > N; f is asymptotically 
bounded below by g, written f{n) = Cl{g{n)), if there exist constants C2 > and N > such that /(n) > C2g{n) for n > N: and / is 
asymptotically bounded above and below by g, written /(n) = ©(g(n)), if C2 g{n) < f{n) < ci gin) for all n > N. 
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(SDP) which can be solved efficiently |25|, | |26) , | (33) . Unfortunately, for random geometric graph^ and grids, 



which are the relevant topologies for large wireless ad-hoc and sensor networks, even the optimized version of 
pairwise gossip is extremely wasteful in terms of communication requirements. For example for a grid topology, 
the number of required messages scales like 0(n^ loge"^) | [25) , ijSSj. Observe that this is of the same order as the 
energy required for every node to flood its estimate to all other nodes. On the contrary, the obvious solution of 
averaging numbers on a spanning tree and flooding back the average to all the nodes requires only 0{n) messages. 
Constructing and maintaining a spanning tree in dynamic and ad-hoc networks introduces significant overhead and 
complexity, but a quadratic number of messages is a high price to pay for fault tolerance. 

D. Faster Gossip Algorithms 

Pairwise gossip converges very slowly on grids and random geometric graphs because of its diffusive nature. 
Information from nodes is essentially performing random walks, and, as is well known, a random walk on the two- 
dimensional lattice has to perform cP steps to cover distance d. One approach to gossiping faster is to modify the 
algorithm so that there is some directionality in the underlying diffusion of information. Assuming that nodes have 
knowledge of their geographic location, we can use a modified algorithm called geographic gossip f35l. The idea 
of geographic gossip is to combine gossip with greedy geographic routing towards a randomly selected location. If 
each node has knowledge of its own location and under some mild assumptions on the network topology, greedy 
geographic routing can be used to build an overlay network where any pair of nodes can communicate. The overlay 
network is a complete graph on which pairwise uniform gossip converges with 8(nloge~^) iterations. At each 
iteration, we perform greedy routing, which costs B(-\/n/ log n) messages on a random geometric graph (also the 
order of the diameter of the network). In total, geographic gossip thus requires 0(?i^ '^ log e^^/^/Iogn) messages. 
The technical part of the analysis involves understanding how this can be done with only local information: assuming 
that each node only knows their own location, routing towards a randomly selected location is not identical to routing 
towards a randomly selected node. If the nodes are evenly spaced, however, these two processes are almost the 
same and the Q{in}-^) message scaling still holds pS) . 

Li and Dai p6) , p7) recently proposed Location-Aided Distributed Averaging (LADA), a scheme that uses 
partial locations and Markov chain lifting to create fast gossiping algorithms. Lifting of gossip algorithms is based 
on the seminal work of Diaconis et al. p8) and Chen et al. ["39] on lifting Markov chain samplers to accelerate 
convergence rates. The basic idea is to lift the original chain to one with additional states; in the context of gossiping, 
this corresponds to replicating each node and associating all replicas of a node with the original. LADA creates 

'^The family of random geometric graphs with n nodes and connectivity radius r, denoted G{n, r), is obtained by placing n nodes uniformly 
at random in the unit square, and placing an edge between two nodes if their Euclidean distance is no more than r. In order to process data in 
the entire network, it is important that the network be connected (i.e., there is a route between every pair of nodes). A fundamental result due 



to Gupta and Kumar |34| states that the critical connectivity threshold for Q{n, r) is rcon(n) = 'hat is, if r does not scale as fast 



as Tconin), then the network is not connected with high probability, and if r scales at least as fast as rcon{n-), then the network is connected 
with high probability. Throughout this paper, when using random geometric graphs it is implied that we are using Q{n, rcon{n)), in order to 
ensure that information flows across the entire network. 
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one replica of a node for each neighbor and associates the policy of a node given it receives a message from 
the neighbor with that particular lifted state. In this manner, LADA suppresses the diffusive nature of reversible 
Markov chains that causes pairwise randomized gossip to be slow. The cluster-based LADA algorithm performs 
slightly better than geographic gossip, requiring <d['n}-^ log e^^/ (log nf"^) messages for random geometric graphs. 
While the theoretical machinery is different, LADA algorithms also use directionality to accelerate gossip, but can 
operate even with partial location information and have smaller total delay compared to geographic gossip, at the 
cost of a somewhat more complicated algorithm. A related scheme based on lifting was proposed concurrently 
by Jung, Shah, and Shin [40 1. Mosk-Aoyama and Shah pTj use an algorithm based on the work of Flajolet and 
Martin | |42) to compute averages and bound the averaging time in terms of a "spreading time" associated with the 
communication graph, with a similar scaUng for the number of messages on grids and RGGs. 

Just as algorithms based on lifting incorporate additional memory at each node (by way of additional states 
in the lifted Markov chain), another collection of algorithms seek to accelerate gossip computations by having 
nodes remember a few previous state values and incorporate these values into the updates at each iteration. These 
memory-based schemes can be viewed as predicting the trajectory as seen by each node, and using this prediction 
to accelerate convergence. The schemes are closely related to shift-register methods studied in numerical analysis to 
accelerate linear system solvers. The challenge of this approach is to design local predictors that provide speedups 
without creating instabilities. Empirical evidence that such schemes can accelerate convergence rates is shown in 
143), and numerical methods for designing linear prediction filters are presented in ||44|, pS). Recent work of 



Oreshkin et al. |46| shows that improvements in convergence rate on par with of geographic gossip are achieved by 
a deterministic, synchronous gossip algorithm using only one extra tap of memory at each node. Extending these 
theoretical results to asynchronous gossip algorithms remains an open area of research. 

The geographic gossip algorithm uses location information to route packets on long paths in the network. One 
natural extension of the algorithm is to allow all the nodes on the routed path to be averaged jointly. This can 
be easily performed by aggregating the sum and the hop length while routing. As long as the information of the 
average can be routed back on the same path, all the intermediate nodes can replace their estimates with updated 
value. This modified algorithm is called geographic gossip with path averaging. It was recently shown pT) that 
this algorithm converges much faster, requiring only <d{^/n) gossip interactions and 9(nloge^^) messages, which 
is clearly minimal. 

A related distributed algorithm was introduced by Savas et al. pS) , using multiple random walks that merge in 
the network. The proposed algorithm does not require any location information and uses the minimal number of 
messages, 0(nlogn), to average on grid topologies with high probability. The coalescence of information reduces 
the number of nodes that update information, resulting in optimal communication requirements but also less fault 
tolerance. In most gossip algorithms all nodes keep updating their information which, as we discuss in the next 
section, adds robustness with respect to changes to the network and noise in communications. 

Finally, we note the recent development of schemes that exploit the broadcast nature of wireless communications in 
order to accelerate gossip rates of convergence 149), pO), either by having all neighbors that overhear a transmission 
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execute a local update, or by having nodes eavesdrop on their neighbors' communication and then using this 
information to strategically select which neighbor to gossip with next. The next section discusses issues arising 
when gossiping specifically over wireless networks. 

III. Rate Limitations in Gossip Algorithms 

Rate limitations are relevant due to the bandwidth restrictions and the power limitations of nodes. Finite trans- 
mission rates imply that nodes learn of their neighbors' states with finite precision; if the distortion is measured by 
the MSB, then it is well established that the operational distortion rate function is exponentially decaying with the 
number of bits |51 1, which implies that the precision doubles for each additional bit of representation. For example, 
in an AWGN channel with path loss inversely proportional to the distance squared, r^, the rate R needs to be below 
the capacity bound R < C = l/21og(l + 7r^^). Then, at a fixed power budget, every bit of additional precision 
requires approximately shrinking the range by half; i.e., fixing 7, the channel capacity increases as the inter-node 
distance decreases. For a uniform network deployment, this would reduce the size of each node's neighborhood 
by about 75%, decreasing the network connectivity and therefore the convergence speed. This simple argument 
illustrates the importance of understanding if the performance of gossip algorithms degrades gracefully as the 
communication rate of each link decreases. 

Before summarizing the key findings of selected literature on the subject of average consensus under communi- 
cation constraints, we explain why some papers care about this issue and some do not. 

A. Are Rate Constraints Significant? 

In most sensor network architectures today, the overhead of packet headers and reliable communication is so 
great that using a few bytes to encode the gossip state variables exchanged leads to negligible additional cost while 
practically giving a precision that can be seen as infinite. Moreover, we can ignore bit errors in transmissions, which 
very rarely go undetected thanks to CRC bits. It is natural to ask: why should one bother studying rate constraints 
at all? 

One should bother because existing sensor network modems are optimized to transmit long messages, infrequently, 
to nearby neighbors, in order to promote spatial bandwidth reuse, and were not designed with decentralized iterative 
computation in mind. Transmission rates are calculated amortizing the overhead of establishing the link over the 
duration of very long transmission sessions. 

Optimally encoding for computation in general (and for gossiping in particular) is an open problem; very few have 
treated the subject of communication for computation in an information theoretic sense (see, e.g., p2) , p3] ) and 
consensus gossiping is nearly absent in the landscape of network information theory. This is not an accident. Broken 
up in parts, consensus gossip contains the elements of complex classical problems in information theory, such as 
multi-terminal source coding, the two-way channel, the feedback channel, the multiple access of correlated sources 
and the relay channel p4) ; this is a frightening collection of open questions. However, as the number of possible 
applications of consensus gossip primitives expands, designing source and channel encoders to solve precisely 
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this class of problems more efficiently, even though perhaps not optimally, is a worthy task. Desired features are 
efficiency in exchanging frequently, and possibly in an optimal order, few correlated bits, and exchanging with 
nodes that are (at least occasionally) very far, to promote rapid diffusion. Such forms of communications are very 
important in sensor networks and network control. 

Even if fundamental limits are hard to derive, there are several heuristics that have been applied to the problem 
to yield some achievable bound. Numerous papers have studied the effects of intermittent or lossy links in the 
context of gossip algorithms (i.i.d. and correlated models, symmetric and asymmetric) ||55|-||64[. In these models, 
lossy links correspond to masking some edges from the topology at each iteration, and, as we have seen above, 
the topology directly affects the convergence rate. Interestingly, a common thread running through all of the work 
in this area is that so long as the network remains connected on average, convergence of gossip algorithms is not 
affected by lossy or intermittent links, and convergence speeds degrade gracefully. 

Another aspect that has been widely studied is that of source coding for average consensus, and is the one that 



we consider next in Section III-B It is fair to say that, particularly in wireless networks, the problem of channel 



coding is essentially open, as we will discuss in section [TlI-CI 
B. Quantized consensus 

Quantization maps the state variable exchanged Xj{t) onto codes that correspond to discrete points Qt.j{xj{t)) = 
<lj{t) G Qt,j C M. The set Qt j is referred to as the code used at time t by node j; the points qj{t) are used to 
generate an approximation Xj{t) of the state Xj{t) G M. that each node needs to transmit; the quantizer rate, in bits, 
is Rt.j — log2 IQt.jl, where |^| is the cardinality of the set A. Clearly, under the constraints specified previously 
on the network update matrix W{t), the consensus states, {cl : c e M}, are fixed points. The evolution of the 
nodes' quantized states is that of an automaton; under asynchronous random exchanges, the network state forms a 
Markov chain with Y[j=i \ Qt,j\ possible states and consensus states {cl : c e M} that are absorbing states. The 
cumulative number of bits that quantized consensus diffuses throughout the network asymptotically is: 



CO n 



t=l t=l j=l 

The first simple question is: for a fixed uniform quantizer with step-size A, i.e., 

Xj{t) ~ vLmA{xj{t)) — argmin |a;j(i) — q\, 

qeQ 

where Q — {0, ±A, ±2A, . . . , ±(2-'^^^ — 1)A}, do the states x{t) always converge (in a probabilistic sense) to the 
fixed points cl? The second is: what is the distortion d ^linife^oo x{t), ^ J2^=i 3;i(0)l^ due to limited R^^i or a total 
budget Rtof l Fig. [T] illustrates the basic answers through numerical simulation. Interestingly, with a synchronous 
gossip update, quantization introduces new fixed points other than consensus (Fig. [TJa)), and asynchronous gossiping 
in general reaches consensus, but without guarantees on the location of outcome (Fig. [TJb)). 



Kashyap et al. 1 65 1 first considered a fixed code quantized consensus algorithm, which preserves the network 



average at every iteration. In their paper, the authors draw an analogy between quantization and load balancing 
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Fig. 1. Quantized consensus over a random geometric graph witli n = 50 nodes transmission radius r = .3 and initial states S [0, 1] witli 
uniform quantization witli 128 quantization levels. Synchronous updates (a) and pairwise exchange (b). 
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among processors, which naturally comes with an integer constraint since the total number of tasks are finite and 
divisible only by integers (see, e.g., | [T9) , [ |66| , ||67[). Distributed policies to attain a balance among loads were 
previously proposed in |68|, 0691. Assuming that the average can be written as ^ X]J=i^i(0) — f denoting 



L = S mod n, under these updates in |65| it is proven that any algorithm meeting the aforementioned conditions 
makes every node converge to either L or L + 1, thereby approximating the average. The random gossip algorithm 
analyzed in fTO) leads to a similar result, where the final consensus state differs at most by one bin from the true 
average; the same authors discuss bounds on the rate of convergence in | |7T| . In these protocols the agents will 
be uncertain on what interval contains the actual average: the nodes whose final value is L will conclude that the 
average is in [L — 1, L + 1] and those who end with L + 1 will think that the average is in [L, L + 2]. Benezit et 



al. 1 72 1 proposed a slight modification of the policy, considering a fixed rate class of quantization strategies that are 
based on voting, requiring only 2 bits of memory per agent and attaining a consensus on the interval that contains 
the actual average. 

To overcome the fact that not all nodes end up having the same quantized value, a simple variant on the quantized 
consensus problem that guarantees almost sure convergence to a unique consensus point was proposed concurrently 



in 1 73 and |74|. The basic idea is to dither the state variables by adding a uniform random var iableu-Z^(-f ,f) 
prior to quantizing the states, i.e., Xi{t) — umA{xi{t) + u). This modest change enables gossip to converge to a 
consensus almost surely, as shown in |75|. This guarantees that the nodes will make exactly the same decision. 
However, the algorithm can deviate more from the actual average than the quantized consensus policies considered 
in | [65| . The advantage of using a fixed code is the low complexity, but with relatively modest additional cost, the 
performance can considerably improve. 

Carli et al. in |76| noticed that the issue of quantizing for consensus averaging has analogies with the problem of 
stabilizing a system using quantized feedback |j77], which amounts to partitioning the state-space into sets whose 
points can be mapped to an identical feedback control signal. Hence, the authors resorted to control theoretic tools to 
infer effective strategies for quantization. In particular, instead of using a static mapping, they model the quantizer 
Qt{xi{t)) at each node i as a dynamical system with internal state £,i{t), which is coupled with the consensus 
update through a quantized error variable qi{t) (see Fig.|2]i. They study two particular strategies. They refer to the 
first as the zoom in - zoom out uniform coder/decoder where they adaptively quantize the state as follows. The 
node states are defined as 

6(0 = (i_i,,(<),/,(i)). (10) 

The quantized feedback and its update are 

X,{t) = = £_!,,; (t) + /,(t)g,(t) ; (11) 
,m = uni.(^ — j, (12) 
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Fig. 2. Quantized consensus Node i Encoder and Node j Decoder, with memory. 



which is basically a differential encoding, and /^(t) is the stepsize, updated according to 

^ h-^^^'^ , (13) 

[ koutMt) if 

which allows the encoder to adaptively zoom-in and out, depending on the range of qi{t). The second strategy has 
the same node states but uses a logarithmic quantizer, 

Mt) = e.(< + i) =e.(<) + g»W; (14) 

q,{t) = \ogs{x,{t)-^,{t)), (15) 



where the logarithmic quantization amounts to: 



1 



q,it) = signix.it) - ^,it)) l^ — j (16) 

In [ [76| numerical results are provided for the convergence of the zoom-in/out quantizer, while the properties of the 
logarithmic quantizer are studied analytically. Remarkably, the authors prove that if the state average is preserved and 
if < (5 < '^^^ , then the network reaches asymptotically exactly the same state as the un-quantized average 

consensus. In other words, for all i, limfe_j.oo a^i (0 — ^ ^"=1 ^« (*-*)■ needs to observe that the logarithmic 
quantizer replaces state values in an uncountable set M with discrete countable outputs £ G N, in the most efficient 
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way | [77) , but there are still infinite many such sets; in other words, the logarithmic quantizer has unlimited range 
and therefore Rt j — oo. Hence, in practice, one will have to accept a penalty in accuracy when its range is limited. 

The vast signal processing literature on sampling and quantization can obviously be applied to the consensus 
problem as well to find heuristics. It is not hard to recognize that the quantizers analyzed in |76j are equivalent to 
predictive quantizers. Noting that the states are both temporally and spatially correlated, it is clear that encoding using 
the side information that is available at both transmitter and receiver can yield improved performance and lower cost; 



this is the tenet of the work in | |78) , |79 1, which analyzed a more general class of quantizers. They can be captured in 
a similar framework as that of |j76j by adding an auxiliary state variable Qj{t), which affects the state of the decoder 
only (see the decoder in Fig.|3]l. The idea is similar, since i_i j(t + 1) in (H} is replaced in |79| by the optimum 
hnear minimum mean-squared error prediction, performed using k previous states x^i,i{t) = X!f=i o,i,k{l)xi(t — I). 
Similarly, the receiver state is introduced to utilize the idea of coding with side information |80) , where the side 
information about Xi{t) that is available at, say, receiver j consists of the receiver's present state Xj{t), as well as 
possibly its own past states and those of neighbors in communication with node j. The decoder augmented state 
iCij (t) T in 1 79 1 is useful to reap the benefits of the refinement in the prediction of Xi{t) that the decoder can 

obtain using its own side information. This prediction x-i^ij{t) can more closely approximate the true state Xi(t) 
compared to the transmitter x_i i{t) and this, in turn, means that ( [T2| i can be replaced by a nested quantizer, such 
as for example the nested lattice quantizers in | [8Tj . In practice, to keep the complexity at bay, one can use a static 
nested lattice quantizer at the transmitter without any memory, while using the current local state as the j-node 
decoder state, i.e., Cij{t) = Xj{t). The main analytical result in f79\ is the conclusion that, even with the lowest 
complexity (i.e. prediction memory k = 1 only or Qj{t) ~ Xj{t) and no memory) one needs finite < oo to 
guarantee that the network will reach consensus with a bounded error d {ivoik^oo x{t), ^ X^ILi 2^^(0)1^ < 
that decreases as a function of R^f. This is useful to establish since one may argue that, as long as the network is 
finite, flooding each value from each node, rather than gossiping, would require a total cost in term of transmission 
bits that is finite, which can also be reduced via optimal joint source and network coding methods. It is meaningful 
to ask if gossiping can also lead to a similar rate-distortion tradeoff and the result in | [79j suggests that this is, 
indeed, the case. 
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Recent work has begun to investigate information-theoretic performance bounds for gossip. These bounds charac- 
terize the rate-distortion tradeoff either (i) as a function of the underlying network topology assuming that each link 
has a finite capacity | [82| , or (ii) as a function of the rate of the information source providing new measurements 
to each sensor |[83). 

C. Wireless channel coding for average consensus 

Quantization provides a source code, but equally important is the channel code that is paired with it. First, using 
the separation of source and channel coding in wireless networks is not optimal in general. Second, and more 
intuitively, in a wireless network there are a variety of rates that can be achieved with a variety of nodes under 
different traffic conditions. The two key elements that determine what communications can take place are scheduling 
and channel coding. Theoretically, there is no fixed-range communication; any range can be reached albeit with 
lower capacity. Also, there is no such thing as a collision; rather, there is a tradeoff between the rate that multiple 
users can simultaneously access the channel. 

The computational codes proposed in [84) aim to strike a near-optimal trade-off for each gossip iteration, by 
utilizing the additive noise multiple access channel as a tool for directly computing the average of the neighborhood. 
The idea advocated by the authors echoes their previous work p3) : nodes send lattice codes that, when added through 
the channel, result in a lattice point that encodes a specific algebraic sum of the inputs. Owing to the algebraic 
structure of the channel codes and the Unearity of the channel, each recipient decodes directly the linear combination 
of the neighbors' states, which provides a new estimate of the network average when added to the local state. The 
only drawbacks of this approach is that 1) it requires channel state information at the transmitter, and 2) that only 
one recipient can be targeted at the time. The scenario considered is closer to that in |83|, since a stream of data 
needs to be averaged, and a finite round is dedicated to each input. The key result proven is that the number of 
rounds of gossip grows as 0{\ogn? /r^) where r is the radius of the neighborhood. 

IV. Sensor Network Applications of Gossip 

This section illustrates how gossip algorithms can be applied to solve representative problems in wireless sensor 
networks. Of course, gossip algorithms are not suited for all distributed signal processing tasks. They have proven 
useful, so far, for problems that involve computing functions that are linear combinations of data or statistics at 
each node. Two straightforward applications arise from distributed inference and distributed detection. When sensors 
make conditionally independent observations, the log-likelihood function conditioned on a hypothesis Hj is simply 
the sum of local log-likelihood functions, ^'^ZP{^i\Hi), and so gossip can be used for distributed detection 

(see also p5) , |[86)). Similarly, if sensor readings can be modeled as i.i.d. Gaussian with unknown mean, distributed 
inference of the mean boils down to computing the average of the sensor measurements, and again gossip can be 
applied. Early papers that made a broader connection are those of Saligrama et al. |86|, and Moallemi and Van 
Roy | [87| , which both discuss connections between gossip algorithms and belief propagation. 
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Below we consider three additional example applications. Section IV-A describes a gossip algorithm for distributed 
linear parameter estimation that uses stochastic approximation to overcome quantization noise effects. Sections IV-B 
and |IV-C] illustrate how gossip can be used for distributed source localization and distributed compression, respec- 
tively. We also note that gossip algorithms have recently been applied to problems in camera networks for distributed 
pose estimation 188), p9). 



A. Robust Gossip for Distributed Linear Parameter Estimation 

The present section focuses on robust gossiping for distributed linear parameter estimation of a vector of 
parameters with low-dimensional observations at each sensor We describe the common assumptions on sensing, the 
network topology, and the gossiping protocols. Although we focus on estimation, the formulation is quite general 
and applies to many inference problems, including distributed detection and distributed locaUzation. 

1) Sensing/Observation Model: Let e M™^^ be an m-dimensional parameter that is to be estimated by a 
network of n sensors. We refer to as a parameter, although it is a vector of m parameters. For definiteness we 
assume the following observation model for the i-th sensor: 

z,{t) ^ H,9 + w,{t) (17) 

where: {^Zi[t) e l^™'^^}t>o '^^e i.i.d. observation sequence for the i-th sensor; {u'i(t)}(>Q is a zero-mean 
i.i.d. noise sequence of bounded variance. For most practical sensor network applications, each sensor observes 
only a subset of rrij of the components of 9, with ^ to. Under such conditions, in isolation, each sensor can 
estimate at most only a part of the parameter Since we are interested in obtaining a consistent estimate of the 
entire parameter 6 at each sensor, we need some type of observability condition. We assume the matrix 

n 

Y^HlH, (18) 

4=1 

is full rank. Note that the invertibility is even required by a centralized estimator (one which has access to data 
from all sensors at all time) to get a consistent estimate of 9. It turns out that, under reasonable assumptions on the 
network connectivity, this necessary condition for centralized observability is sufficient for distributed observability, 
i.e., for each sensor to obtain a consistent estimate of 0. It is not necessary to restrict to time-invariant observation 
matrices and the i/^s can be random time-varying | [90| , as would be required in most regression based analyses. In 
general, the observations need not come from a linear statistical model and may be distributions parameterized by 
9. The distributed observability would then correspond to asymptotic distinguishability of the collection of these 
distributions over the network. A generic formulation in such a setting requires the notion of separably estimable 
observation models (see [91 1). 

An equivalent formulation of the estimation problem in the setting considered above, comes from the distributed 
least mean square (LMS) adaptive filtering framework |[92)-|[94). The objective here is slightly different. While we 
are interested in consistent estimates of the entire parameter at each sensor, the LMS formulations require, in a 
distributed way, to adapt to the environment to produce a desired response at each sensor, and the observability issue 



DIMAKIS ETAL. : GOSSIP ALGORITHMS FOR DISTRIBUTED SIGNAL PROCESSING 



17 



is not of primary importance. A generic framework for distributed estimation, both in the static parameter case and 
when the parameter is non-stationary, is addressed in |95|. An important aspect of algorithm design in these cases 
is the choice of the inter-sensor weight sequence for fusing data or estimates. In the static parameter case, where 
the objective is to drive all the sensors to the true parameter value, the weight sequence necessarily decays over 
time to overcome the accumulation of observation and other forms of noises, whereas, in the dynamic parameter 
estimation case, it is required that the weight sequence remains bounded away from zero, so that the algorithm 
possesses tracking abilities. We direct the reader to the recent article 1%) for a discussion along these lines. In the 



dynamic case, we also suggest the significant literature on distributed Kalman filtering (see, e.g., |[97|-[ 101 1 and 
the references therein), where the objective is not consensus seeking among the local estimates, but, in general, 
optimizing fusion strategies to minimize the mean-squared error at each sensor. 

It is important to note here that average consensus is a specific case of a distributed parameter estimation model, 
where each sensor initially takes a single measurement, and sensing of the field thereafter is not required for the 
duration of the gossip algorithm. Several distributed inference protocols (for example, p4) , 1 102 1, 1 103| ) are based 
on this approach, where either the sensors take a single snapshot of the field at the start and then initiate distributed 



consensus protocols (or more generally distributed optimization, as in |103|) to fuse the initial estimates, or the 
observation rate of the sensors is assumed to be much slower than the inter-sensor communicate rate, thus permitting 
a separation of the two time-scales. 

2) Distributed linear parameter estimation: We now briefly discuss distributed parameter estimation in the linear 



observation model ( 17 1. Starting from an initial deterministic estimate of the parameters (the initial states may be 
random, we assume deterministic for notational simplicity), Xi{Q) E M™^^, each sensor generates, by a distributed 
iterative algorithm, a sequence of estimates, {xi{t)}^^^. To simplify the discussion in this section, we assume a 
synchronous update model where all nodes exchange information and update their local estimates at each iteration. 
The parameter estimate Xi{t + 1) at the i-th sensor at time t + 1 is a function of: 1) its previous estimate; 2) the 
communicated quantized estimates at time t of its neighboring sensors; and 3) the new observation Zi{t). The data is 
subtractively dithered quantized, i.e., there exists a vector quantizer Q{.) and a family, {i'j-j (i)}, of i.i.d. uniformly 
distributed random variables on [—A/2, A/2) such that the quantized data received by the i-th sensor from the j-th 
sensor at time t is Q{xj{t) + Vij{t)), where Vij{t) — [v}j{t), ■ ■ ■ It then follows that the quantization 

error, eij{t) E M™^^, is a random vector, whose components are i.i.d. uniform on [—A/2, A/2) and independent 

of Xj{t). 

3) Stochastic approximation algorithm: Let Afi{t) denote the neighbors of node i at iteration t; that is, j £ Ni{t) 
if i can receive a transmission from j at time t. In this manner, we allow the connectivity of the network to vary 
with time. Based on the current state, Xi{t), the quantized exchanged data {Q{xj{t) + Vij{t))} ^^j^ ^^.y and the 
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observation Zi{t), the updated estimate at node i is 

+ = x,{t)-a{t) b {x^{t) (19) 

- Q{x,{t) + v,,{t)))-Hj {z,{t) 

- H,x,{t)) 

In ( 19 1, 6 > is a constant and {ck(i)}j>Q is a sequence of weights satisfying the persistence conditiorj^ 

a(i)>0, ^Q:(i)=oo, Ya'^{t)<oo (20) 
t t 

Algorithm ( [T9| is distributed because for sensor n it involves only the data from the sensors in its neighbor- 
hood Afi{t). 

The following result from f9T) characterizes the desired statistical properties of the distributed parameter es- 
timation algorithm just described. The flavor of these results is common to other stochastic approximation algo- 
rithms | |104| . First, we have a law of large numbers-like result which guarantees that the estimates at each node 
will converge to the true parameter estimates, 

Pf lim = e, yi) = 1 (21) 

If, in addition to the conditions mentioned above, the weight sequence is taken to be 

(22) 

for some constant a > 0, we also obtain a central limit theorem-like result, describing the distribution of estimation 
error over time. Specifically, for a sufficiently large, we have that the error, 

Vi{x{t) - 1 (g) 6*) 

converges in distribution to a zero-mean multivariate normal with covariance matrix that depends on the observation 
matrices, the quantization parameters, the variance of the measurement noise, Wi{t), and the constants a and b. The 
two most common techniques for analyzing stochastic approximation algorithms are stochastic Lyapunov functions 
and the ordinary differential equations method p04| . For the distributed estimation algorithm ([T9|, the results just 
mentioned can be derived using the Lyapunov approach | |9T| , | |105) . 

Performance analysis of the algorithm for an example network is illustrated in Figure |4] An example network 
of n = 45 sensors are deployed randomly on a 25 x 25 grid, where sensors communicate in a fixed radius and 
are further constrained to have a maximum of 6 neighbors per node. The true parameter 9* £ ]R*^. Each node is 
associated with a single component of 9* . For the experiment, each component of 9* is generated by an instantiation 
of a zero mean Gaussian random variable of variance 25. (Note, the parameter 6* here has a physical significance 



^We need the a{t) to sum to infinity, so tliat tiie algoritlim 'persists' and does not stop; on tlie other hand, the a sequence should be square 
summable to prevent the build up of noise over time. 
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and may represent the state of the field to be estimated. In this example, the field is assumed to be white, stationary 
and hence each sample of the field has the same Gaussian distribution and independent of the others. More generally, 
the components of 9* may correspond to random field samples, as dictated by the sensor deployment, representing a 
discretization of the PDE governing the field.) Each sensor observes the corresponding field component in additive 
Gaussian noise. For example, sensor 1 observes zi{t) — 6\ +wi{t), where wiit) ~ A/'(0, 1). Clearly, such a model 
satisfies the distributed observability condition 

G^Yl = I = G-^ (23) 

i 

(Note, here Hi — ef, where is the standard unit vector with 1 at the i-th component and zeros elsewhere.) 
Fig. Qa) shows the network topology, and Fig. |4|b) shows the normalized error of each sensor plotted against the 
iteration index t for an instantiation of the algorithm. The normalized error for the i-th sensor at time t is given 
by the quantity ||xi(<) — 0*\\ /45, i.e., the estimation error normalized by the dimension of 9*. We note that the 
errors converge to zero as established by the theoretical findings. The decrease is rapid at the beginning and slows 
down at t increases. This is a standard property of stochastic approximation based algorithms and is attributed to 
the decreasing weight sequence a{t) required for convergence. 

It is interesting to note that, although the individual sensors suffer from low rank observations of the true parameter, 
by collaborating, each of them can reconstruct the true parameter value. The asymptotic normality shows that the 
estimation error at each sensor decays as l/Vi, the decay rate being similar to that of a centralized estimator having 
access to all the sensor observations at all times. The efficiency of the distributed estimator is measured in terms of 
its asymptotic variance, the lower limit being the Fisher information rate of the corresponding centralized estimator. 
As expected, because of the distributed nature of the protocol (information needs to disseminate across the entire 
network) and quantized (noisy) inter-sensor communication, the achieved asymptotic variance is larger than the 
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centralized Fisher information rate. In the absence of quantization (perfect communication), it can be shown that 



the parameter a in eqn. ( 22 1 can be designed appropriately so that the asymptotic variance of the decentralized 
estimator matches the centralized Fisher information rate, showing that the distributed estimator described above is 
efficient for the centralized estimation problem. An example of interest, with Gaussian observation noise is studied 
in | [9T) , where it is shown that asymptotic variance attainable by the distributed algorithm is the same as that of 
the optimum (in the sense of Cramer-Rao) centralized estimator having access to all information simultaneously. 
This is an interesting result, as it holds irrespective of the network topology. Such a phenomenon is attributed to 
a time scale separation between the consensus potential and the innovation rate (rate of new information entering 
the network), when inter-sensor communication is unquantized (perfect) with possible link failures. 

As noted before, the observation model need not be linear for distributed parameter estimation. In | [9T| a large class 
of nonlinear observation models were considered and a notion of distributed nonlinear observability called separably 
estimable observable models introduced. Under the separably estimable condition, there exist local transforms under 
which the updates can be made linear However, such a state transformation induces different time scales on the 



consensus potential and the innovation update, giving the algorithm a mixed time-scale behavior (see | [9T) , |106| 
for details.) This mixed time-scale behavior and the effect of biased perturbations leads to the inapplicability of 
standard stochastic approximation techniques. 

B. Source Localization 

A canonical problem, encompassing many of the challenges which commonly arise in wireless sensor network 
applications, is that of estimating the location of an energy-emitting source |[l|. Patwari et al. 1 107| presents an 



excellent overview of the many approaches that have been developed for this problem. The aim in this section 
is to illustrate how gossip algorithms can be used for source localization using received signal strength (RSS) 
measurements. 

Let 6* G denote the coordinates of the unknown source, and for i = 1, . . . , n, let ?/i e denote the location 
of the ith sensor. The RSS measurement at node i is modeled as 

a 

m - 

where a > is the signal strength emitted at the source, (3 is the path-loss coefficient, and Wj is additive 
white Gaussian noise. Typical values of f3 are between 2 and 4. This model was validated experimentally in 
P08[. Centralized maximum likelihood estimators for single and multiple-source scenarios based on this model are 



presented in 1 109 1 and | |1 10| . Because the maximum likelihood problem is, in general, non-linear and non-convex, 
it is challenging to solve in a decentralized fashion. Distributed approaches based on finding a cyclic route through 
the network are presented in 19), pO). 



An alternative approach, using gossip algorithms 1 1 1 1), forms a location estimate 9 by taking a linear combination 



of the sensor locations weighted by a function of their RSS measurement, 



EtiKih) ' ^^^^ 



DIMAKIS ETAL. : GOSSIP ALGORITHMS FOR DISTRIBUTED SIGNAL PROCESSING 



21 



where K : M+ — > M+ is a monotone increasing function satisfying K{Q) = and limj_j.oo K{f) < oo. Intuitively, 
nodes that are close to the source measure high RSS values, so their locations should be given more weight than 
nodes that are further away, which will measure lower RSS values. Taking K{f) = 1{^>^}, where 7 > is a 
positive threshold, and where is the indicator function, ( |25] l reduces to 

^ ELl^'l{||;;,-9||<7-^/'^} .^^s 
fl = ^F=^n ] , UO) 

2^1=1 -'-{||yi-e||<7-i//3} 

which is simply the centroid of the locations of sensors that are no further than 7"^/^ from the source. In | 111) , it 



was shown that this estimator benefits from some attractive properties. First, if the sensor locations yi are modeled 
as uniform and random over the region being sensed, then di is a consistent estimator, as the number of sensors 
grows. It is interesting to note that one does not necessarily need to know the parameters a 01 (5 precisely to 
implement this estimator. In particular, because ( |25| ) is self-normalizing, the estimator automatically adapts to the 
source signal strength, a. In addition, [1 1 1| shows that this estimator is robust to choice of 7. In particular, even if 
/? is not known precisely, the performance of (|25| degrades gracefully. On the other hand, the maximum likelihood 
approach is very sensitive to model mismatch and estimating a and (3 can be challenging. 

Note that ( |25] l is a ratio of linear functions of the measurements at each node. To compute p5| ), we run two 
parallel instances of gossip over the network, one each for the numerator and the denominator. If each node 
initializes xf{0) = yiK{fi), and xf{0) — K{fi), then executing gossip iterations will cause the values at each 
node to converge to limt^.oo a;f (t) = ^ J2]=i Vj^ifj) limj^oo xf = ^ K{fj). Of course, in a practical 

implementation one would stop gossiping after a fixed number of iterations, tgtop, which depends on the desired 
accuracy and network topology. Then, each node can locally compute the estimate xf {t stop)/ xf {t stop) of the 
source's location. Note that throughout this section it was assumed that each node knows its own location. This 
can also be accomplished using a gossip-style algorithm, as described in 1 112| . 



C. Distributed Compression and Field Estimation 

Extracting information in an energy-efficient and communication-efficient manner is a fundamental challenge in 
wireless sensor network systems. In many cases, users are interested in gathering data to see an "image" of activity 
or sensed values over the entire region. Let /j e M denote the measurement at node i, and let / G M" denote 
the network signal obtained by stacking these values into a vector Having each sensor transmit fi directly to an 
information sink is inefficient in many situations. In particular, when the values at different nodes are correlated 
or the signal is compressible, then one can transmit less data without loosing the salient information in the signal. 
Distributed source coding approaches attempt to reduce the total number of bits transmitted by leveraging the 
celebrated results of Slepian and Wolf 1 113| to code with side information p 14) , 1 115| . These approaches make 



assumptions about statistical characteristics of the underlying data distribution that may be difficult to verify in 
practice. 

An alternative approach is based on linear transform coding, gossip algorithms, and compressive sensing. It has 
been observed that many natural signals are compressible under some linear transformation. That is, although / 
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Fig. 5. Example illustrating compression of a smooth signal. Panel (a) shows the original smooth signal which is sampled at 500 random 
node locations, and nodes are connected as in a random geometric graph. Panel (b) illustrates the m-term approximation error decay in both 
the original basis and using the eigenvectors of the graph Laplacian as a transform, which is analogous to taking a Fourier transform of signals 
supported on the network. Panel (c) illustrates the reconstruction error after gossiping on random linear combinations of the sensor measurements 
and reconstructing using compressed sensing techniques. Note that using more random linear projections (larger k), gives lower error, but the 
number of projections used is much smaller than the network size. 



may have energy in all locations (i.e., fi>0 for all i), there is a linear basis transformation matrix, T e M"^", such 
that when / is represented in terms of the basis T by computing 9 — Tf, the transformed signal 6 is compressible 
(i.e., 9j « for many j). For example, it is well known that smooth one-dimensional signals are well approximated 
using the Fourier basis, and piece-wise smooth images with smooth boundaries (a reasonable model for images) 



are well-approximated using wavelet bases |116|. 

To formally capture the notion of compressibiUty using ideas from the theory of nonlinear approximation 1 1 17) , 
we reorder the coefficients 9j in order of decreasing magnitude, 

1^)1 > 1^(2)1 >|e(3)|>--->|0(„)|, (27) 

and then define the best m-term approximation of / in T as = X^Jli where Xj denotes the jth 

column of T. This is analogous to projecting / onto the m-dimensional subspace of T that captures the most energy 
in /. We then say that / is a-compressible in T, for a > 1, when the mean squared approximation error behaves 
like 

-|l/-/^™^ll'<Cm-2", (28) 
n 

for some constant C > 0. Since the error exhibits a power-law decay in m for compressible signals, it is possible 
to achieve a small mean squared approximation error while only computing and/or communicating the few most 
significant coefficients 6'(i), . . . ,9(j^y Figure |5] shows an example where 500 nodes forming a random geometric 
graph sample a smooth function. As a compressing basis T, we use the eigenvectors of the normalized graph 
Laplacian (a function of the network topology), which are analogous to the Fourier basis vectors for signals 
supported on G 1 ,118] , 



DIMAKIS ETAL. : GOSSIP ALGORITHMS FOR DISTRIBUTED SIGNAL PROCESSING 



23 



Observe that each coefficient, 9j, is a linear function of the data at each node, and so one could conceivably 
compute these coefficients using gossip algorithms. Assuming that each node i knows the values {Tj.;}"^! in 
each basis vector, to compute 6j, we can initialize Xi{Q) = nTj^ifi, and, by gossiping, each node will compute 
limt^oo Xi{t) = X]fc=i Tj,kfk = (^j- The main challenge with this approach is that the indices of the most significant 
coefficients are very signal-specific, and are generally not known in advance. 

We can avoid this issue by making use of the recent theory of compressive sensing ]! 19) - p2T| , which says 
that one can recover sparse signals from a small collection of random linear combinations of the measurements. In 
the present setting, to implement the gathering of k compressive sensing measurements using gossip algorithms, 
each node initializes k parallel instances of gossip with a:ij(0) — nAijfi, j — l,...,k, where Ai,j, e.g., are 
i.i.d. zero-mean normal random variables with variance 1/n. Let Xj denote the limiting value of the jth gossip 
instance at each node. Stacking these into the vector, x, any node can recover an estimate of the signal / by solving 
the optimization 

n 

mm\\x - AT'^ef +T^\9i\, (29) 
1=1 

where t > is a regularization parameter. In practice, the values Ai j can be pseudo-randomly generated at each 
node using a predefined seeding mechanism. Then, any user can retrieve the gossip values {xij{t)}^^-^ from any 
node i and solve the reconstruction. Moreover, note that the compressing transformation T only needs to be known at 
reconstruction time, and to initialize the gossip instances each node only needs its measurement and pseudo-randomly 
generated values Ai,j. In general, there is a tradeoff between 1) fc, the number of compressed sensing measurements 
collected, 2) e, the accuracy to which the gossip algorithm is run, 3) the number of transmissions required for this 
computation, and 4) the average reconstruction accuracy available at each node. For an a-compressible signal /, 
compressed sensing theory provides bounds on the mean squared reconstruction error as a function of k and a, 
assuming the values, x, are calculated precisely. Larger k corresponds to lower error, and the error decays rapidly 
with k (similar to the m-term approximation), so one can obtain a very accurate estimate of / with fc <C n 
measurements. Inaccurate computation of the compressed sensing values, x, due to gossiping for a finite number of 
iterations, can be thought of as adding noise to the values x, and increases the overall reconstruction error. Figure |5|c) 
illustrates, via numerical simulation, the tradeoff between varying k and the number of gossip iterations. For more 
on the theoretical performance guarantees achievable in this formulation, see ]I22[ , | |123| . A gossip-based approach 



to solving the reconstruction problem in a distributed fashion is described in 1 124|. For an alternative approach to 



using gossip for distributed field estimation, see |I25| 



V. Conclusion and Future Directions 

Because of their simplicity and robustness, gossip algorithms are an attractive approach to distributed in-network 
processing in wireless sensor networks, and this article surveyed recent results in this area. A major concern in 
sensor networks revolves around conserving limited bandwidth and energy resources, and in the context of iterative 
gossip algorithms, this is directly related to the rate of convergence. One thread of the discussion covered fast 
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gossiping in wireless network topologies. Another thread focused on understanding and designing for the effects 
of wireless transmission, including source and channel coding. Finally, we have illustrated how gossip algorithms 
can be used for a diverse range of tasks, including estimation and compression. 

Currently, this research is branching into a number of directions. One area of active research is investigating 
gossip algorithms that go beyond computing linear functions and averages. Just as the average can be viewed as the 
minimizer of a quadratic cost function, researchers are studying what other classes of functions can be optimized 



within the gossip framework |126|. A related direction is investigating the connections between gossip algorithms 
and message-passing algorithms for distributed inference and information fusion, such as belief propagation p7) , 
\\21\ . While it is clear that computing pairwise averages is similar to the sum-product algorithm for computing 
marginals of distributions, there is no explicit connection between these families of distributed algorithms. It would 
be interesting to demonstrate that pairwise gossip and its generalizations correspond to messages of the sum- product 
(or max-product) algorithm for an appropriate Markov random field. Such potentials would guarantee convergence 
(which is not guaranteed in general iterative message-passing) and further establish explicit convergence and message 
scheduling results. 

Another interesting research direction involves understanding the effects of intermittent links and dynamic 
topologies, and in particular the effects of node mobility. Early work 1 128| has analyzed i.i.d mobility models 



and shown that mobility can greatly benefit convergence under some conditions. Generalizing to more realistic 
mobility models seems to be a very interesting research direction that would also be relevant in practice since 
gossip algorithms are more useful in such dynamic environments. 

Gossip algorithms are certainly relevant in other applications that arise in social networks and the interaction 
of mobile devices with social networks. Distributed inference and information fusion in such dynamic networked 
environments is certainly going to pose substantial challenges for future research. 
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